-
Notifications
You must be signed in to change notification settings - Fork 13.7k
server/public_simplechat vision (basic ok), toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) #17142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hanishkvc
wants to merge
308
commits into
ggml-org:master
Choose a base branch
from
hanishkvc:hkvc_server_simplechat_vision_v0500
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
server/public_simplechat vision (basic ok), toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) #17142
hanishkvc
wants to merge
308
commits into
ggml-org:master
from
hanishkvc:hkvc_server_simplechat_vision_v0500
+4,845
−478
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also possible refinement wrt trapping, if needed, added as comment all or allSettled to use or not is the question. whether to wait for a round trip through the related event loop or not is also a question.
Dont forget to map members of got entity from fetch to things from saved original promise, bcas remember what is got is a promise. also add some comments around certain decisions and needed exploration
A basic go at it
Declare the result of UrlReq as a DataClass, so that one doesnt goof up wrt updating and accessing members. Duplicate UrlRaw into UrlText, need to add Text extracting from html next for UrlText
As _UrlopenRet not exposed for use outside urllib, so decode and encode the data. Add skeleton to try get the html/xml tree top elements
As html can be malformed, xml ElementTree XMLParser cant handle the same properly, so switch to the HtmlParser helper class that is provided by python and try extend it. Currently a minimal skeleton to just start it out, which captures only the body contents.
First identify lines which have only whitespace and replace them with lines with only newline char in them. Next strip out adjacent lines, if they have only newlines
Ensures that if the url being requested as any query strings in them then things dont get messed up, when the url to get inc its query is extracted from the proxy request's query string
also now track header, footer and nav so that they arent captured
Add a new send headers common helper and use the same wrt the overridden send_error as well as do_OPTIONS This ensures that if there is any error during proxy opertions, the send_error propogates to the fetch from any browser properly without browser intercepting it with a CORS error
ie opening db as well as a transaction to access a store within the db.
Added put and get helpers wrt indexedDB. Updated save and load related logics in SimpleChatTCRV.
Allow passing the accept list for file type input element helper. Inturn given that currently it is used for the image selection for vision models, set it to jpeg and png in the caller for the same.
Contributor
Author
|
Changed Chat session auto save and user optional load to use indexedDB instead of localStorage, so that there is enough space availabe for saving chat sessions, given that images are in the mix now. Cleaned up the ui a tiny little bit |
Had forgotten to move the one shot resp into try catch before. Fixed it. Ensure both oneshot multipart resp try catch Add some todos for later. Add a new check wrt response being normal or a error related one ie the content is actually a error message.
Trap the mouseenter and mouseleave events wrt the chatmessage block to show the chatmessage-popover ui inturn within the corresponding chatmessage. When ever user clicks the popover menu's delete button, the uniqId associated with each chat message, is now used to delete the user selected chat message. Initial skeleton wrt clipboard copy.
Use position fixed followed by using position-area so that one can use anchor passed through showPopover. Avoid the hidePopover from mouseLeave, because as the mouse moves over the poped up popover ui, the mouseLeave will be called on the chat message block/container. And that would have closed the popover and inturn mouseEnter would have got called on ChatMsg, given now with the popover hidden, the ChatMsg block is directly accessible. So this leads to twinkling / toggling ie continues show/hide of the popover. Inturn in mouseEnter itself decide when to close the previous popover (ie if the chatmessage id is different from what it was before).
As position-area is not yet officially supported in firefox (its only in nightly builds, as of now), so switching to the inset block/inline start/end css properties.
Currently only the text/content part is handled wrt copy. Image URL isnt handled, even if present.
Rename to get_chatmessage_index. Return the index of the chatmessage deleted. Add a scroll element into view helper. Now the UniqId wrt the ChatMessage is stored as a attribute wrt the corresponding chatmessage block. And same used wrt scrolling after deleting messages. When a chat message is deleted, previously the screen would have moved to last message in the chat session, now it tries to remain wrt adjacent messages on the screen.
This ensures that the tool call edit/trigger is reshown, when user submits a empty response and the logic stops from going ahead.
avoid refreshing the full chat session messages ui also Update readme
If deleting a non last (and non just before last) message, then just directly remove the corresponding chat message block from the ChatSession UI and be done with it. However if deleting the last (or just before last) message, then one needs to decide on whether tool call edit/trigger ui is shown or removed and so... and similarly wrt tool response edit/submit.
Make the toolresp non happy path use a direct insert chat msg wrt ui, rather than recreating the full chat session ui fresh.
Rename lastMsg to prevLastMsg to match semantic better Return seperate flags to indicate add and show success or not Update tool response happy path to use this optimised add show instead of chat_show (which recreates the full chat session ui).
Now even the user response handling flow (handle_user_submit) has been updated to use the optimised chat show logic which adjusts / updates the chat session ui wrt the last two messages only and not the full chat history in the chat session. Move the logic wrt last two messages and ui, into its own func and inturn use it everywhere as required.
Needed as some places we require to delete the chat message as well as update the corresponding ui block. And in other places we require to just remove the ui block from chat session ui and not the underlying chat message from chat history. Change ui updateprev appendlast to use the ui remove logic.
SC.add discards any temp role message and not just tool temp msg New SC.add_smart which either adds a new message to the chat session or replaces existing last message in the chat session, based on same role or different, given that the chat session cant have the same role type chat message occuring adjacent in it. Inturn rename MCUI chatmsg_add_uishow to chatmsg_addsmart_uishow and use add_smart * helps wrt trying to rerun a tool call with modified args or so. Rather prev discard temp role messages in SC.add not good enough as uniqId will change in its case. Helps avoid adding duplicate ToolTemp messages in chat session ui * can help if loading a prev chat session which ended in a user message without a ai response. User can type in a new message and continue that old chat session, with the new message, replacing the old user message as well as initiating handshake with ai server in a proper manner. Replace MCUI chatmsg_ui_updateprev_appendlast with chatmsg_ui_refresh which is a more generic flow, which takes care of updating the ui as needed irrespective of if the specified set of messages are already in the chat session ui or not. Also allows the caller to control how many messages at the end need to be refreshed wrt ui.
Rename chatmsg_ui_refresh to chat_uirefresh. Even in case of the delete path and inturn deleting one of the last two messages in a chat session, now use the generified chat uirefresh logic instead of the chat_show full session refresh / recreation of the session ui Inturn to make the uirefresh really generic and usable in all cases including above case, now take care of clearing the tool call edit / trigger at the beginning, so that the last 2 messages decide in turn whether to show a tool call edit/trigger ui or not, as well as the tool call response edit / submit ui.
also update some notes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the initial basic skeleton for supporting vision models going forward to the tools/server/public_simplechat alternate web client ui. Basic tested with Gemma3 and Qwen3VL
This builds on the previous PR in this series ie #17038 which completed the current initial go at tool calling which also includes a set of client side based builtin and bundled tool calls for code execution, data storage, web access, web search, pdf and xml use, ... as well as support for looking into reasoning/chainofthought of the models which share the same.